Display control apparatus, display control method and recording medium

ABSTRACT

A display control apparatus displays a cursor based on an operation performed by an operator on a screen of a display device. The display controlling unit corrects the in-screen position of the instruction image based on a relationship between a first position indicated by the hand having a first shape and a second position indicated by the hand having a second shape in a case where a second operation is detected after a first operation is detected and displays the instruction image at the in-screen position corresponding to the second position.

BACKGROUND

Technical Field

The present disclosure relates to a display control apparatus, a displaycontrol method, and a recording medium for controlling a displayposition of an instruction image based on an operation performed by anoperator.

Description of the Related Art

A method has been proposed in which an operator operating an apparatusperforms a specific operation (hereinafter, called a “gesture”) near adisplay device by watching an operation screen displayed on the displaydevice so that the operator can operate the apparatus. Japanese PatentLaid-Open No. 2013-200654 discloses a display control apparatus whichdetects a movement of a hand of an operator and moves a cursor displayedon a display device in accordance with the detected movement of thehand. Japanese Patent Laid-Open No. 2014-99184 discloses a computerwhich detects a shape and a movement of a hand of an operator andexecutes a process corresponding to the shape and movement of the handof the operator.

However, in the past, because the central position of a hand isidentified as the position of the hand, it may be recognized that theposition of the hand has changed when the shape of the hand is changed.As a result, the display position of the cursor is changed when theoperator has changed the shape of the hand, which may reduce theoperationality. For example, when a first operation for operating with ahand closed into a first is changed to a second operation for operatingwith a tip of the index finger only stretched from the closed hand, thedisplay position of a cursor may possibly changed against the intentionof the operator, which may make user's operations difficult.

SUMMARY OF THE INVENTION

An aspect of the present invention provides a display control apparatusincluding an operation identifying unit configured to identify a type ofoperation based on a shape of a hand of an operator, and a displaycontrolling unit configured to display an instruction image at anin-screen position corresponding to a first position indicated by thehand having a first shape in a case where a first operation performed bythe hand having the first shape is detected and displays an instructionimage at an in-screen position corresponding to a second positionindicated by the hand having a second shape in a case where a secondoperation performed by the hand having the second shape is detected,wherein the display controlling unit corrects the in-screen position ofthe instruction image based on a relationship between the first positionand the second position in a case where the second operation is detectedafter the first operation is detected and displays the instruction imageat the in-screen position corresponding to the second position.

Another aspect of the present invention provides a display controlmethod including identifying a type of operation based on a shape of ahand of an operator, a first display step for displaying an instructionimage at an in-screen position corresponding to a first positionindicated by the hand having a first shape in a case where a firstoperation performed by the hand having the first shape is detected, andcorrecting the in-screen position of the instruction image based on arelationship between the first position and a second position differentfrom the first position indicated by the hand having a second shapeindicating the second position and a second display step for displayingthe instruction image at an in-screen position corresponding to thesecond position in a case where a second operation performed by the handhaving the second shape is detected after the first display step.

Another aspect of the present invention provides a computer-readablerecording medium storing a program for causing a computer to executeidentifying a type of operation based on a shape of a hand of anoperator, a first display step for displaying an instruction image at anin-screen position corresponding to a first position indicated by thehand having a first shape in a case where a first operation performed bythe hand having the first shape is detected, and correcting thein-screen position of the instruction image based on a relationshipbetween the first position and a second position different from thefirst position indicated by the hand having a second shape indicatingthe second position and a second display step for displaying theinstruction image at an in-screen position corresponding to the secondposition in a case where a second operation performed by the hand havingthe second shape is detected after the first display step.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a first diagram schematically illustrating a configuration ofa display system S according to a first exemplary embodiment andillustrating a display state as a result of a first operation.

FIG. 1B is a second diagram schematically illustrating a configurationof a display system S according to a first exemplary embodiment andillustrating a display state as a result of a second operation.

FIG. 2A is a first diagram schematically illustrating a configuration ofthe display system S according to the first exemplary embodiment andillustrating a display state as a result of the first operation.

FIG. 2B is a second diagram schematically illustrating a configurationof the display system S according to the first exemplary embodiment andillustrating a display state as a result of the second operation.

FIG. 3 is a block diagram illustrating a configuration of a displaycontrol apparatus according to the first exemplary embodiment.

FIG. 4A is a first diagram illustrating a shape of a hand.

FIG. 4B is a second diagram illustrating a shape of a hand.

FIG. 5A is a first diagram illustrating positional information regardinga hand and a positional information regarding a finger.

FIG. 5B is a second diagram illustrating positional informationregarding a hand and a positional information regarding a finger.

FIG. 6 illustrates an exemplary hand-shape/gesture correspondence table.

FIG. 7A is a first diagram illustrating a method for calculatingcursor-position correction information.

FIG. 7B is a second diagram illustrating the method for calculatingcursor-position correction information.

FIG. 8 illustrates an exemplary gesture-operation conversion table.

FIG. 9 is a flowchart illustrating an operating procedure of the displaysystem S.

FIG. 10A is a first diagram illustrating a still image acquired by adisplay image acquiring unit.

FIG. 10B is a second diagram illustrating a still image acquired by thedisplay image acquiring unit.

FIG. 10C is a third diagram illustrating a still image acquired by thedisplay image acquiring unit.

FIG. 10D is a fourth diagram illustrating a still image acquired by thedisplay image acquiring unit.

FIG. 11 is a flowchart illustrating an operating procedure from start ofimage-capturing by an imaging apparatus.

FIG. 12 is a flowchart illustrating an operating procedure for drawingan image to be displayed on a display device by a drawing unit.

FIG. 13A is a first diagram illustrating an operation for shifting animage to be displayed on a display screen.

FIG. 13B is a second diagram illustrating an operation for shifting animage to be displayed on a display screen.

FIG. 14 illustrates an operation for shifting an image to be displayedon a display screen.

FIG. 15 illustrates a gesture/offset correspondence table usable by acursor position correcting unit according to a second exemplaryembodiment.

FIG. 16 illustrates a mapping expression table usable by the cursorposition correcting unit according to the second exemplary embodiment.

FIG. 17 is a flowchart illustrating an operating procedure according tothe second exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS First Exemplary Embodiment Outline ofDisplay System S

FIG. 1A is a first diagram schematically illustrating a configuration ofa display system S according to a first exemplary embodiment andillustrating a display state as a result of a first operation. FIG. 1Bis a second diagram schematically illustrating a configuration of thedisplay system S according to the first exemplary embodiment andillustrating a display state as a result of a second operation. Adisplay system S includes an imaging apparatus 100, a display controlapparatus 200, and a display device 300.

The imaging apparatus 100 may be a camera, for example, and outputscaptured image data regarding an imaging range 101 to the displaycontrol apparatus 200.

The display control apparatus 200 detects a gesture of an operator 102at a position apart from the display device 300 based on the capturedimage data acquired from the imaging apparatus 100 and displays aninstruction image at a position corresponding to the gesture on thedisplay screen 301 of the display device 300. The instruction image maybe a graphic image to be displayed on the display screen 301 based on anoperation performed by the operator 102 and may be a cursor 302, forexample.

The display control apparatus 200 determines whether the captured imagedata includes the operator 102 or not. If it is determined that thecaptured image data includes the operator 102, an operation range 103 isdetermined based on human figure information describing a contour of thebody of the operator 102. The display control apparatus 200 identifiesthe positions of the shoulders, arms and hands of the operator 102 bypattern matching, for example, and determines the operation range 103having a size equal to or smaller than a range in which the operator 102is capable of moving his or her hand based on the identified positions.In this case, the display control apparatus 200 determines such that theshape of the operation range 103 is a rectangle similar to the shape ofthe display screen 301 and has sides parallel to the sides of thedisplay screen 301.

The display control apparatus 200 detects a movement and a shape of apredetermined region (such as a hand 104 in FIG. 1A and a hand 105 inFIG. 1B) of the body and determines the type of gesture based on achange of the detected movement and shape to identify the type ofoperation performed by the operator 102. The display control apparatus200 controls the content displayed on the display screen 301 on thedisplay device 300 in accordance with the type of operation.

More specifically, when the operator 102 moves his hand 104 having aclosed state from outside the operation range 103 to inside theoperation range 103, the display control apparatus 200, as illustratedin FIG. 1A, displays the cursor 302 at a position within the displayscreen 301 corresponding to the position of the hand 104 in theoperation range 103. In this state, the relationship between the rangeof the display screen 301 and the position of the cursor 302 is equal tothe relationship between the operation range 103 and the position of thehand 104. When the operator 102 moves the hand 104 from left to rightwithin the operation range 103, the display control apparatus 200controls the display state of the cursor 302 such that the cursor 302displayed on the display screen 301 can move from left to right.

As illustrated in FIG. 1B, when the operator 102 moves his or her hand105 having a finger 106 stretched from the closed hand from outside theoperation range 103 to inside the operation range 103, the displaycontrol apparatus 200 displays the cursor 302 at a position on thedisplay screen 301 corresponding to the position of the finger 106 inthe operation range 103. When the operator 102 moves the finger 106 fromleft to right within the operation range 103, the display controlapparatus 200 controls the display state of the cursor 302 such that thecursor 302 displayed on the display screen 301 can move from left toright.

FIG. 2A is a first diagram schematically illustrating a configuration ofthe display system S according to the first exemplary embodiment andillustrating a display state as a result of the first operation. FIG. 2Aillustrates an identical state to that in FIG. 1A. In other words, FIG.2A illustrates a state in which the operator 102 moves the hand 104having a closed state from outside the operation range 103 to inside theoperation range 103.

FIG. 2B is a second diagram schematically illustrating a configurationof the display system S according to the first exemplary embodiment andillustrating a display state as a result of the second operation. FIG.2B illustrates a display state of the cursor 302 when the operator 102changes the shape of the hand 105 to a state that the finger 106 isstretched to point in the state illustrated in FIG. 2A. In this case,though the position of the finger 106 in the operation range 103 isidentical to the position illustrated in FIG. 1B, the position of thecursor 302 on the display screen 301 is identical to the positionillustrated in FIG. 2A rather than the position illustrated in FIG. 1B.In this manner, when the operator 102 performs a predetermined operationfor changing the shape of the hand 104, such as stretching the finger106 at a position of the cursor 302 determined by the closed hand 104,the display control apparatus 200 according to this exemplary embodimentprevents the position of the cursor 302 from moving, which may improvethe operationality.

Configuration of Display Control Apparatus 200

A configuration of the display control apparatus 200 will be describedin detail below.

FIG. 3 is a block diagram illustrating a configuration of the displaycontrol apparatus 200 according to the first exemplary embodiment. Thedisplay control apparatus 200 has a data storage unit 201, a displayimage acquiring unit 202, a display control unit 203, a captured imageacquiring unit 204, an operation range identifying unit 205, ahand/finger identifying unit 206, a hand/finger information determiningunit 207, a gesture identifying unit 208, a cursor position correctingunit 209, and an operation identifying unit 210.

The display control apparatus 200 has a CPU, a ROM, and a RAM, notillustrated. The ROM stores a display control program to be executed bythe CPU. The CPU executes the display control program stored in the ROMto control operations of the display control apparatus 200 by using theRAM as a working memory. The ROM stores tables to be used by the CPU forcontrolling the display control apparatus 200.

The data storage unit 201 stores still images to be displayed on thedisplay screen 301. The data storage unit 201 may be a hard disk, forexample.

The display image acquiring unit 202 acquires a plurality of stillimages from the data storage unit 201, generates still image dataacquired by adding index information indicative of the order of imagesto the still images, and outputs the still image data to the displaycontrol unit 203.

The display control unit 203 causes an image to be displayed on thedisplay device 300 based on the still image data input from the displayimage acquiring unit 202. The display control unit 203 synthesizes abutton and the cursor 302 usable for operating to change the image to bedisplayed with a still image for drawing, generates display image data,and outputs the display image data to the display device 300 to displaythe image. Details of the display control unit 203 will be describedbelow.

The captured image acquiring unit 204 acquires captured image dataincluding an image acquired by capturing the hand 104 of the operator102 from the imaging apparatus 100 and outputs the acquired capturedimage data to the operation range identifying unit 205.

The operation range identifying unit 205 identifies the operation range103 based on the captured image data input from the captured imageacquiring unit 204 and generates operation range information oprincluding image data acquired by cutting out the operation range 103from the captured image data. The operation range identifying unit 205then outputs the operation range information opr to the hand/fingeridentifying unit 206.

For example, the operation range identifying unit 205 may identify aposition sp (xs, ys) of the shoulder and a length la of the arm of theoperator 102 included in the captured image data input from the capturedimage acquiring unit 204 based on body shape information, contourinformation, or silhouette information of many people prestored in theROM, for example, and generate human figure information which isinformation describing the position of the shoulder and the position ofthe arm.

Next, the operation range identifying unit 205 determines the operationrange 103 based on the position sp (xs, ys) of the shoulder and thelength la of the arm included in the human figure information. Morespecifically, the operation range identifying unit 205 calculates thecoordinates (upper left op_lu, lower left op_ld, upper right op_ru,lower right op_rd) of four corners of the operation range 103 by usingthe following expressions.

op_lu(x,y)=(xs−la,ys+la×0.5)

op_ld(x,y)=(xs−la,ys−la×0.5)

op_ru(x,y)=(xs+la,ys+la×0.5)

op_rd(x,y)=(xs+la,ys−la×0.5)

The operation range identifying unit 205 extracts image data of theoperation range 103 from the captured image data based on the calculatedcoordinates and generates the operation range information opr. Theoperation range identifying unit 205 outputs the generated operationrange information opr to the hand/finger identifying unit 206. Accordingto this exemplary embodiment, image data included in the operation rangeinformation opr generated by the operation range identifying unit 205correspond to a rectangular image of 160 pixels×90 pixels deep.

The hand/finger identifying unit 206 identifies the shape of the handbased on the open/closed states of a plurality of fingers of theoperator 102. More specifically, the hand/finger identifying unit 206may compare the operation range information opr and shape information,contour information, or silhouettes of many fingers prestored in theROM, for example, to identify the contour of the hand and fingers. Thehand/finger identifying unit 206 determines whether fingers arestretched or not based on the identified contours of the hand andfingers to generate hand information h.

The hand information h is a structure including information describingthe presence/absence of the hand and the thumb, index, middle, ring, andlittle fingers. When the hand has a “closed state” in which the hand isnot stretched as indicated by the hand 104 illustrated in FIG. 4A, thehand information h contains hand 1, thumb 0, index finger 0, middlefinger 0, ring finger 0, and little finger 0. When the state of the handhas a “point state” in which the index finger is stretched as indicatedby the hand 104 and finger 106 in FIG. 4B, the hand information hcontains hand: 1, thumb: 0, index finger: 1, middle finger: 0, ringfinger: 0, and little finger: 0. In the hand information h, theinformation corresponding to a stretched finger is 1 while informationcorresponding to an unstretched finger is 0. The hand/finger identifyingunit 206 outputs the generated hand information h and the operationrange information opr to the hand/finger information determining unit207.

The hand/finger information determining unit 207 identifies a positionof the hand 104 having a first shape in which the operator 102 does notstretch a specific finger as a first position and a position of thefinger 106 of the hand 105 having a second shape in which the operator102 stretches a specific finger (such as the finger 106) as a secondposition. The hand/finger information determining unit 207 determinespositional information regarding the hand and positional informationregarding the fingers within the operation range information opr basedon the operation range information opr and hand information h, forexample. FIG. 5A is a first diagram illustrating positional informationregarding the hand and positional information regarding a fingerdetermined by the hand/finger information determining unit 207. FIG. 5Bis a second diagram illustrating positional information of the hand andpositional information regarding the fingers determined by thehand/finger information determining unit 207.

When the hand information h describes that no finger is stretched, thehand/finger information determining unit 207 only determines positionalinformation hp (xp, yp) of the hand as illustrated in FIG. 5A. Thepositional information hp (xp, yp) of the hand describes a position of aregion in the hand 104 and may be a central position of the hand 104,for example.

When the hand information h describes that one of the fingers isstretched, the hand/finger information determining unit 207 determinesthe positional information hp (xp, yp) of the hand and positionalinformation fp (xi, yi) as illustrated in FIG. 5B. In the exampleillustrated in FIG. 5B, because the index finger is stretched, thehand/finger information determining unit 207 determines the positionalinformation fp (xi, yi) of the finger based on the position of the tipof the index finger. The hand/finger information determining unit 207outputs the positional information hp of the hand and the positionalinformation fp of the finger to the display control unit 203. Thehand/finger information determining unit 207 outputs the handinformation h, positional information hp of the hand, and positionalinformation fp of the finger to the gesture identifying unit 208.

The gesture identifying unit 208 identifies the type of gesture based onthe shape and position of the hand of the operator 102 in a capturedimage. The gesture identifying unit 208 identifies the type of gesturecorresponding to the shape of the hand based on the hand information hinput from the hand/finger information determining unit 207 and ahand-shape/gesture correspondence table stored in the ROM. FIG. 6illustrates an example of the hand-shape/gesture correspondence table.

In the hand-shape/gesture correspondence table illustrated in FIG. 6,information describing whether the hand of the operator 102 is includedin the operation range 103, information describing whether any of thefingers is stretched or not, and the type of gesture are associated. Agesture “none” in the hand-shape/gesture correspondence table indicatesa state in which the hand is not present within the operation range 103.A gesture “closed” indicates a state in which the hand is closed. Agesture “open” indicates a state in which all fingers are stretched adthe hand is opened. A gesture “point” indicates a state in which theindex finger is only stretched and the finger tip of the index finger isexecuting a process.

The gesture identifying unit 208 identifies gesture information ges(n)describing a gesture corresponding to the shape of the hand everypredetermined time period and stores the identified gesture informationges(n) and the hand information h(n) indicating the shape of the hand atthat time point in association in the RAM. The gesture identifying unit208 outputs the gesture information ges(n) to the display control unit203. The gesture information ges(n) and h(n) is stored in the RAM asarrangement data where n is an integer indicating a position in thearrangement data.

When the immediately preceding gesture information ges(t−1) and thelatest gesture information ges(t) have a predetermined relationship, thegesture identifying unit 208 determines that the gesture has changed andoutputs a gesture determination result “detected” and the positionalinformation hp of the hand and positional information fp of the fingerto the cursor position correcting unit 209. In this case, t is aninteger indicating a time when the gesture identifying unit 208identifies gesture information ges.

When the immediately preceding gesture information ges(t−1) is “closed”and the latest gesture information ges(t) is “point”, the gesture changeresult “detected” indicating that the gesture has changed from “closed”to “point” is notified to cursor position correcting unit 209. Thegesture identifying unit 208 outputs a gesture determination result“none” when the gesture identifying unit 208 determines that the gesturehas not been changed. Because the central position does not change whenthe immediately preceding gesture information ges(t−1) is “closed” andthe latest gesture information ges(t) is “open”, the gesture identifyingunit 208 may output the gesture determination result “none”.

The cursor position correcting unit 209 calculates cursor-positioncorrection information based on the gesture change result, positionalinformation hp of the hand and positional information fp of the fingerinput from the gesture identifying unit 208. FIGS. 7A and 7B illustratea method for calculating the cursor-position correction information.

As illustrated in FIG. 7A, the position in the operation range 103 ofthe hand 104 corresponding to the gesture “closed” is (xp, yp). In thisstate, the position of the cursor 302 in the display screen 301 is (xc,yc). Here, xc and yc are calculated by the following expressions.

xc=xp×the number of horizontal pixels of the display screen 301÷thenumber of horizontal pixels of opr

yc=yp×the number of vertical pixels of the display screen 301÷the numberof vertical pixels of opr

When the positional information hp (xp, yp) of the hand is (40, 15), thenumber of horizontal pixels of the display screen 301 is equal to 1920,the number of vertical pixels of the display screen 301 is equal to1080, the number of horizontal pixels of opr is equal to 160, the numberof vertical pixels of opr is equal to 90, xc=40×1920÷160=480, andyc=15×1080÷90=180.

A case will be described in which the operator stretches his/her indexfinger from the state in FIG. 7A to the state in FIG. 7B. When the tippositional information (xi, yi) of the index finger is (60, 50), theposition (xc′, yc′) in the display screen 301 corresponding to the tipposition of the index finger is (720, 600) so that the position of thecursor 302 can move from the position in 7A.

Here, when the gesture change result input from the gesture identifyingunit 208 is “detected”, the cursor position correcting unit 209calculates cursor-position correction information cpc (offset_x,offset_y) based on the positional information hp of the hand andpositional information fp of the finger by using the followingexpressions.

offset_x=(xi−xp)×the number of horizontal pixels of the display screen301÷the number of horizontal pixels of opr offset_y=(yi−yp)×the numberof vertical pixels of the display screen 301÷the number of verticalpixels of opr

As described above, when (xp, yp)=(40, 15) and (xi, yi)=(60, 50), thecursor position correcting unit 209 calculates as offset_x=240 andoffset_y=420. The cursor position correcting unit 209 outputs thecursor-position correction information cpc calculated as described aboveto the cursor control unit 211.

The operation identifying unit 210 identifies the type of operationbased on the shape and position of the hand of the operator 102. Morespecifically, the operation identifying unit 210 identifies the type ofoperation corresponding to the gesture based on the gesture informationges, positional information hp of the hand identified based on the shapeof the hand of the operator 102, which are input from the gestureidentifying unit 208, and gesture-operation conversion table. Theoperation identifying unit 210 notifies the operation information opedescribing the identified type of operation to the display control unit203.

FIG. 8 illustrates an exemplary gesture-operation conversion table. Inthe gesture-operation conversion table, the gesture information ges andthe type of operation are associated. Referring to FIG. 8, the gesture“closed” corresponds to an operation for moving a cursor, the gesture“point” corresponds to an operation for pressing down a button at aposition corresponding to the position of the hand, and the gesture“open” corresponds to an operation for moving an image display position.

The display control unit 203 has a cursor control unit 211, a drawingunit 212 and an image output unit 213. The display control unit 203displays the cursor 302 at a position (xc, yc) corresponding to a firstposition (xp, yp) on the hand 104 having the first shape in a capturedimage based on the first operation performed by the hand 104 having thefirst shape identified by the operation identifying unit 210. Afterthat, when a second operation having a predetermined relationship withthe first operation performed by the hand 105 having the second shape ata second position (xi, yi) different from the first position isdetected, the display control unit 203 continuously displays the cursor302 at the position (xc, yc) within a screen corresponding to the firstposition based on the relationship between the first position and thesecond position. The first position (xp, yp) may be the central positionof the hand 104, for example, and the second position (xi, yi) may bethe position of the tip of the finger 106 of the hand 105, for example.

On the other hand, the display control unit 203 displays the cursor 302at the position (xc, yc) within a screen corresponding to the firstposition (xp, yp) based on the first operation performed by the hand 104having the first shape identified by the operation identifying unit 210,and, after that, when an operation not having a predeterminedrelationship with the first operation is detected, displays the cursor302 at a position within the screen corresponding to the position of thehand performing the operation. For example, when an operation isperformed by the hand 104 having a closed state and then the operator102 opens the hand, the display control unit 203 moves the cursor 302 tothe position within a screen corresponding to the position of the openhand.

Operations to be performed by components of the display control unit 203will be described.

The cursor control unit 211 corrects an in-screen position correspondingto the second position by using an in-screen difference corresponding tothe difference between the position (xp, yp) of the hand 104 and theposition (xi, yi) of the finger 106 in the operation range 103, which isdetermined based on the ratio between the number of pixels of theoperation range 103 and the number of pixels of the display screen 301to generate cursor positional information cp describing the correctedposition. The cursor control unit 211 then notifies the cursorpositional information cp to the display control unit 203 so that thecursor 302 can be continuously displayed at the position (xc, yc) withinthe display screen 301 corresponding to the position of the hand 104.

The cursor control unit 211 calculates the cursor positional informationcp based on the positional information hp of the hand and positionalinformation fp of the finger input from the hand/finger informationdetermining unit 207 and the cursor-position correction information cpcinput from the cursor position correcting unit 209. First, based on thepositional information hp (xp, yp) of the hand and the positionalinformation fp (xi, yi) of the fingers, the cursor control unit 211first calculates a position (xc′, yc′) of the finger before corrected bythe following expression.

xc′=xi×the number of horizontal pixels of the display screen 301÷thenumber of horizontal pixels of opr

yc′=yi×the number of vertical pixels of the display screen 301+thenumber of vertical pixels of opr

Next, the cursor control unit 211 calculates the cursor positionalinformation cp (xc, yc) by correcting the position (xc′, yc′) of thefinger with the cursor-position correction information cpc (offset_x,offset_y).

xc=xc′−offset_x

yc=yc′−offset_y

In this case, as illustrated in FIG. 7B, when (xp, yp)=(40, 15), (xi,yi)=(60, 50), and (offset_x, offset_y)=(240, 420), (xc′, yc′)=(720, 600)are acquired. Thus, cursor positional information cp (xc, yc)=(480, 180)is acquired. In other words, the position of the cursor 302 is identicalto the position of the cursor 302 illustrated in FIG. 7A.

The drawing unit 212 generates display image data based on still imagedata, operation information ope and cursor positional information cp.More specifically, the drawing unit 212 synthesizes an image based onthe still image data, an operation image corresponding to the type ofoperation described in the operation information ope, and an instructionimage corresponding to the cursor 302 at a position described in thecursor positional information cp. The drawing unit 212 identifies thetype of operation intended by the operator 102 based on the operationinformation ope and the cursor positional information cp. The drawingunit 212 changes the display image based on the identified type ofoperation. The drawing unit 212 outputs the display image data generatedby the synthesis to the image output unit 213.

The image output unit 213 may include an HDMI (registered trademark)transmitter. The image output unit 213 outputs the display image datainput from the drawing unit 212 to the display device 300.

As described above, when a second operation “PRESS BUTTON” correspondingto the operator 102 stretching the finger 106 is detected, the displaycontrol unit 203 displays the cursor 302 at the in-screen position (xc,yc) acquired by correcting the in-screen position (xc′, yc′)corresponding to the position of the finger 106 based on the difference(offset_x, offset_y) between the first position (position of the hand104) and the second position. As a result, also when the operator 102stretches the finger 106, the cursor 302 can be continuously displayedat the in-screen position (xc, yc) corresponding to the first position(xp, yp).

Flowchart of Operations of Display System S

FIG. 9 is a flowchart illustrating an operating procedure of the displaysystem S. FIG. 9 illustrates an operating procedure from the time whenthe imaging apparatus 100, display control apparatus 200 and displaydevice 300 are powered on to the time when imaging is started by theimaging apparatus 100.

The display control apparatus 200 monitors whether the power supply isturned on or not (S11). When the power supply is turned on, it triggersthe display image acquiring unit 202 to acquire still image data savedin the data storage unit 201 (S12). FIG. 10A to FIG. 10C illustratestill images corresponding to three pieces of still image data acquiredby the display image acquiring unit 202. The display image acquiringunit 202 gives index information to the images in the order ofacquisition and outputs the still image data to the drawing unit 212.

Next, the drawing unit 212 draws a still image having 0 as the indexinformation, and an operation image is synthesized to the still image,as illustrated in FIG. 10D (S13). The drawing unit 212 generates displayimage data of vertical 1920×horizontally 1080 pixels based on thesynthesized image and outputs them to the image output unit 213. Theimage output unit 213 displays the input display image data to thedisplay device 300 (S14). FIG. 10D illustrates an image acquired bysynthesizing an operation image “NEXT IMAGE” for receiving an operationfor shifting to the next image and an operation image “PREVIOUS IMAGE”for receiving an operation for shifting to the previous image.

Next, the imaging apparatus 100 starts imaging, which is triggered bycompletion of display of the display image data in step S13 (S15). Afterthe imaging apparatus 100 starts imaging, the operator 102 can operatethe imaging apparatus 100 by moving the hand 104 within the operationrange 103.

Next, with reference to FIG. 11 and FIG. 12, operations will bedescribed for controlling the display position of the cursor 302 by thedisplay control apparatus 200 after the imaging apparatus 100 startsimaging. FIG. 11 is a flowchart describing an operating procedure afterthe imaging apparatus 100 starts imaging. FIG. 12 is a flowchartillustrating an operating procedure for drawing an image to be displayedon the display device 300 by the drawing unit 212.

In order to describe operations for changing the image to be displayedon the display screen 301, the operations to be performed by the displaycontrol apparatus 200 will be described with reference to FIGS. 13A and13B and FIG. 14. Operations will be sequentially described in a state(hereinafter, called a “first state”) in which the operator 102 moveshis or her closed hand to the operation range 103, as illustrated inFIG. 13A, a state (hereinafter, called a “second state”) in which theoperator 102 stretches the index finger as illustrated in FIG. 13B afterthe first state, and a state (hereinafter, called a “third state”) inwhich the operator 102 moves the stretched index finger after the secondstate.

Operations in First State

The captured image acquiring unit 204 acquires captured image data (S21)and outputs the captured image data to the operation range identifyingunit 205. Here, the operator 102 has his or her hand closed asillustrated in FIG. 13A.

Next, the operation range identifying unit 205 identifies the positionof a human figure in the input captured image data to determine theoperation range 103, generates operation range information opr acquiredby cutting out the image corresponding to the operation range 103 fromthe captured image data, and outputs it to the hand/finger identifyingunit 206 (S22). The hand/finger identifying unit 206 identifies thecontours of the hand and fingers based on the input operation rangeinformation and generates hand information h (S23). When the hand isclosed as illustrated in FIG. 13A, the hand/finger identifying unit 206generates the hand information h containing hand: 1, thumb: 0, indexfinger: 0, middle finger: 0, ring finger: 0, and little finger: 0.

Next, the hand/finger information determining unit 207 generates thepositional information hp (xp, yp) of the hand within the operationrange information opr based on the positional information fp of thefinger hand information h (S24). Here, when the hand is closed asillustrated in FIG. 13A, the positional information fp of the fingerexhibits no value.

Next, the gesture identifying unit 208 identifies the type of gesturebased on the hand information h input from the hand/finger identifyingunit 206 and the hand-shape/gesture correspondence table (S25). Here,because the input hand information h contains hand: 1, thumb: 0, indexfinger: 0, middle finger: 0, ring finger: 0, and little finger: 0 whenthe hand is closed, the gesture information ges is “closed”. The gestureidentifying unit 208 sets the type of gesture to gesture informationges(n1) and hand information h to hand information h(n1) and stores themin the RAM. The gesture identifying unit 208 outputs the gestureinformation ges to the operation identifying unit 210.

The operation identifying unit 210 identifies the operation informationope based on the gesture information ges input from the gestureidentifying unit 208 and the gesture-operation conversion table todetermine the type of operation (S26). In this case, because the gestureinformation ges is “closed”, the operation identifying unit 210identifies that the operation information ope is “MOVE CURSOR”.

Next, the gesture identifying unit 208 determines whether therelationship between the immediately preceding input gesture informationges(t−1) and the latest gesture information ges(t) indicates apredetermined gesture change or not (S27). Here, the informationdescribing a predetermined gesture change is stored in the ROM. Thepredetermined gesture change is exemplified by a case where the gestureinformation ges(t−1) is “closed” and the gesture information ges(t) is“point”.

When the gesture information ges is changed from “closed” to “point”,the gesture identifying unit 208 generates a gesture change result“detected” and moves to step S28 where correction information isgenerated. When the gesture information ges(t−1) and gesture information(t) are “point”, a gesture change result “continue” is generated. Then,the processing moves to step S28 in which correction information isgenerated. On the other hand, when the gesture information(t) is “none”,it is not included in the predetermined gesture change. Then, theprocessing moves to step S29 without generating correction information.

Here, in the operation in the first state, gesture information ges(n 0)being an initial value of gesture information ges exhibits no value, and“none” is set as the initial value. When the gesture information ges ischanged from “none” to “closed”, the cursor position does not requirecorrection. Therefore, the gesture change result in the state in FIG.13A is “none”, and the gesture identifying unit 208 determines that thepredetermined gesture change does not occur (No in S27).

Next, the cursor control unit 211 calculates the cursor positionalinformation cp (xc, yc) based on the positional information hp of thehand input from the hand/finger information determining unit 207 anddetermines the cursor position (S29).

Next, with reference to the flowchart in FIG. 12, the operations to beperformed by the drawing unit 212 will be described.

The drawing unit 212 identifies a command instructing the type ofoperation by the operator 102 based on the operation information opeinput from the operation identifying unit 210 and the cursor positionalinformation cp input from the cursor control unit 211 (S31). Next, thedrawing unit 212 determines whether the command is an image changingcommand or not (S32).

If the operation information ope indicates “MOVE CURSOR”, the drawingunit 212 determines that the image changing command does not occur (Noin S32). The processing then moves to S34 where the still image isdisplayed (S34). The drawing unit 212 may synthesize an operation imagefor changing an image to be displayed for performing a cursor movingoperation to the in-screen position corresponding to the position of thehand 104 on the first still image illustrated in FIG. 10A, asillustrated in FIG. 13A, for example, and draws the synthesized image onthe display screen 301 (S35). The drawing unit 212 synthesizes thecursor 302 to the first still image and draws it on the display screen301 (S36). Next, the drawing unit 212 outputs the image acquired bysynthesizing the still image, the operation image and the cursor 302 tothe image output unit 213 (S37). The display control apparatus 200repeats the processing from S11 to S37 until the display controlapparatus 200 is powered off.

Operations in Second State

Next, as illustrated in FIG. 13B, processing in a case where theoperator 102 stretches the index finger will be described. If it isdetermined in step S38 in FIG. 12 that the power supply is not turnedoff, the processing returns to S21 in FIG. 11. The processing in stepS21 and step S22 is the same as the processing to be performed in thefirst state in which the operator 102 has his or her hand closed.

After step S22, the hand/finger identifying unit 206 generates handinformation h containing hand: 1, thumb: 0, index finger: 1, middlefinger: 0, ring finger: 0, and little finger: 0 (S23). Next, thehand/finger information determining unit 207 generates positionalinformation hp (xp, yp) of the hand 104 and positional information fp ofthe finger within the operation range information opr based on the handinformation h (S24).

Next, the gesture identifying unit 208 determines that the gestureinformation ges is “point” based on the hand information h input fromthe hand/finger identifying unit 206 hand the hand-shape/gesturecorrespondence table (S25). The operation identifying unit 210determines that the operation information ope is “PRESS BUTTON” based onthe gesture information ges input from the gesture identifying unit 208and the gesture-operation conversion table.

Next, because the immediately preceding input gesture informationges(n1) is “closed” and the latest gesture information ges(n 2) is“point”, the gesture identifying unit 208 determines that apredetermined gesture change has occurred and outputs a gesture changeresult “detected” (Yes in S27). The gesture identifying unit 208 outputsthe gesture change result, the positional information hp of the hand,and positional information fp of the finger to the cursor positioncorrecting unit 209.

If the gesture change result input from the gesture identifying unit 208is “detected”, the cursor position correcting unit 209 calculates thecursor-position correction information cpc based on the positionalinformation hp of the hand and positional information fp of the finger(S28). The cursor position correcting unit 209 notifies the calculatedcursor-position correction information cpc to the cursor control unit211. The cursor control unit 211 calculates the cursor positionalinformation cp (xc, yc) based on the positional information fp of theindex finger and the cursor-position correction information cpc todetermine the cursor position (S29).

Next, processing illustrated in FIG. 12 is performed. If a buttonpressing operation with the stretched finger 106 is detected in a statein which an operation image for changing the display image is displayed,the drawing unit 212 draws a display image to be displayed on thedisplay screen 301 in accordance with the position of the cursor 302 andin response to the button pressing operation. In this case, the drawingunit 212 draws the display image to be shifted when the operation image(such as a “NEXT IMAGE” button) displayed on the display screen 301 isselected for display on the display screen 301.

More specifically, the drawing unit 212 determines that the “PRESSBUTTON” operation has been performed based on the operation informationope (S31). The drawing unit 212 determines that the cursor 302 is overthe “NEXT IMAGE” button based on the cursor positional information cp.From the result, the drawing unit 212 determines that an image changingcommand has occurred (Yes in S32) and changes the index of the imagebeing displayed to change the still image to be displayed (S33).

Next, the drawing unit 212 draws the still image after the change, theoperation image and the cursor 302 (S34 to S36) and outputs the drawsimages to the image output unit 213 (S37). Thus, the images asillustrated in FIG. 13B are displayed. Because the display position ofthe cursor 302 does not change from the cursor moving operation to theimage changing operation, the operator 102 can operate withoutstrangeness.

Operations in Third State

Next, processing will be described in a case where the operator 102moves the hand 105 having the finger 106 stretched from the stateillustrated in FIG. 13B to the right to shift to the state illustratedin FIG. 14. If it is determined in step S38 in FIG. 12 that the powersupply is not turned off, the processing returns to S21 in FIG. 11. Theprocessing from step S21 to step S25 is the same as the processing to beperformed in the second state.

After step S25, because the immediately previously input gestureinformation ges(n 2) is “point” and the latest gesture information ges(n3) is “point”, the gesture identifying unit 208 determines that nogesture change has occurred and the gesture is continuing (No in S27).

If the gesture change result input from the gesture identifying unit 208is “continue” indicating that the gesture is continuing, the cursorposition correcting unit 209 uses the cursor-position correctioninformation cpc of the second state in step S28. The cursor positioncorrecting unit 209 notifies the cursor-position correction informationcpc to the cursor control unit 211. The cursor control unit 211calculates the cursor positional information cp (xc, yc) based on thepositional information fp of the index finger and the cursor-positioncorrection information cpc to determine the cursor position (S29).

Next, advancing to the processing illustrated in FIG. 12, if the secondoperation for button pressing by stretching the finger 106 in the secondstate is detected, the drawing unit 212 draws the cursor 302 at thein-screen position corresponding to the position acquired by correctingthe position of the hand 105 based on the difference between theposition of the hand 104 and the position of the finger 106 in theoperation range 103 if the hand 105 with the finger 106 stretched ismoved. The drawing unit 212 further determines that the “PRESS BUTTON”operation has been performed based on the operation information ope(S31). The drawing unit 212 determines that the cursor 302 is prevent onthe “PREVIOUS IMAGE” button as illustrated in FIG. 14 based on thecursor positional information cp. From the result, the drawing unit 212determines that an image changing command has occurred (Yes in S32) andchanges the index of the image being displayed to change the still imageto be displayed (S33).

Next, the drawing unit 212 draws the still image after the change, theoperation image and the cursor 302 (S34 to S36) and outputs the drawsimages to the image output unit 213 (S37). Thus, the images asillustrated in FIG. 14 are displayed. With the display control apparatus200 according to this exemplary embodiment, when the operator 102 movesthe hand 104 with the index finger stretched and even when theorientation of the hand 105 of the operator 102 or the angle of thefinger 106 changes, the cursor 302 can be continuously displayed at aposition which is not strange to the operator 102.

When the operator 102 opens the hand 104 or moves the hand 104 tooutside the operation range 103 in the state illustrated in FIG. 14, thedisplay control unit 203 finishes the cursor position correction. Forexample, when an operation performed by the closed hand 104 is detectedat a third position different from a first position after the cursor 302is displayed at the first position, the display control unit 203displays an instruction image at the in-screen position corresponding tothe third position based on the operation performed by the hand 105 withthe finger 106 stretched.

Variation Examples

Having described that the operation range identifying unit 205determines the operation range 103 from the position of the shoulder andthe length of the arm of the operator 102, the operation range 103 maybe fixed to a specific space coordinate position. The operation rangeidentifying unit 205 may determine the operation range 103 and then movethe operation range 103 in accordance with a movement of the operator102.

Having described above that detection of the operator 102 in thecaptured image data included in the operator 102 triggers identificationof the operation range 103 by the operation range identifying unit 205,detection of a specific gesture performed by the operator 102 maytrigger identification of the operation range 103 for prevention ofmalfunctions.

Having described above a case where one operator performs an operation,if plurality of operators 102 is identified, the display controlapparatus 200 may give an operation right to the hand 104 of the firstlyidentified operator 102 and identify a gesture based on the shape of thehand 104 of the firstly identified operator 102.

The display control apparatus 200 may display a plurality of cursors 302corresponding to a plurality of hands of a plurality of operators 102 ora plurality of hands of one operator 102 to enable operations using theplurality of hands. For example, in a case where a second hand isincluded in the operation range 103, the display control unit 203displays a second cursor 302 corresponding to the second hand at thein-screen position corresponding to the position corrected based on adifference between the position of the second hand and positions of thehand 104 and finger 106. In this case, the display control apparatus 200may give an operation right to a hand firstly identified within theoperation range 103, for example, and identifies the other hand forassisting the operation.

In this case, when the gesture identifying unit 208 detects a specificgesture change from a pointing gesture of the right hand to a pointinggesture of both hands, the cursor position correcting unit 209 maycorrect the position of the cursor 302 corresponding to the left hand byusing a difference value between positional information of a finger ofthe left hand and positional information of a finger of the right handas an offset. Thus, in response to a gesture performed by a plurality ofhands, the cursor 302 can be displayed at a proper position.

Having described above that the shape of the cursor 302 is an arrow, thecursor 302 may have a hand shape representing open and closed states offive fingers, for example. In this case, the cursor control unit 211 mayfurther correct the cursor position based on a difference value betweenthe tip position of the index finger and the palm position of the cursor302 such that the tip position of the index finger of the hand-shapecursor 302 can be placed at the in-screen position corresponding to thehand 105 of the operator 102 and the operator 102 can operate furtherintuitively.

Having described above that the imaging apparatus 100 captures theoperator 102 in an X-Y plane, the imaging apparatus 100 may be providedunder the operator 102 so that the hand 104 of the operator 102 can becaptured in an X-Z plane, for example. In this case, in order to measurea Z coordinate thereof, a stereo camera may be used as the imagingapparatus 100, and the operation range identifying unit 205 may identifythe operation range 103 in a three-dimensional space based on a distanceimage acquired by triangulation. The operation range identifying unit205 may use a result of a measurement of a Z coordinate by using adistance sensor.

Having described above that the operation identifying unit 210identifies a type of operation based on the gesture-operation conversiontable illustrated in FIG. 8, the correspondence relationship betweengestures and types of operation is arbitrary. For example, an open statemay correspond to moving a cursor, a shift from an open state to aclosed state may correspond to dragging of an image, a closed state maycorrespond to moving an image and enlargement and reduction of size ofthe image, and a point state may correspond to drawing on an image. Thedisplay control apparatus 200 may change the correspondence relationshipdynamically.

According to the configuration described above, a region may sometimesoccur in which the cursor 302 is disabled to move by the amountcorresponding to the cursor-position correction information cpc forcorrecting the cursor position. In this case, the operation rangeidentifying unit 205 may provide a larger range for identifying thepositions of a hand and a finger of an operator by a size correspondingto the cursor-position correction information cpc, for example, than theoperation range 103 to prevent occurrence of a region in which thecursor 302 is disabled to move.

Having described above that the display control apparatus 200 displays astill image on the display device 300, the display control apparatus 200may display a moving image on the display device 300. Having describedabove that the display of the cursor 302 is only controlled based on agesture, a mouse and a keyboard may additionally be used to control thecursor 302.

Having described above that the display control apparatus 200 has thedata storage unit 201, the data storage unit 201 may be providedexternally to the display control apparatus 200. For example, thedisplay image acquiring unit 202 may acquire an image from a USB memoryor may acquire an image from a server connected thereto over a network.

Effects of First Exemplary Embodiment

As described above, in the display control apparatus 200 according tothe first exemplary embodiment, the gesture identifying unit 208identifies the type of gesture based on the shape of a hand and theposition of the hand of the operator in a captured image, and theoperation identifying unit 210 identifies the type of operationcorresponding to the type of gesture. In a case where, after the displaycontrol unit 203 displays the cursor 302 at the in-screen positioncorresponding to the position of the hand 104, if a predeterminedoperation is detected from the stretched finger 106, the cursor 302 canbe displayed continuously without moving the position of the cursor 302displayed in accordance with the position of the hand 104. Thus, whenthe operator 102 operates without an intension to move the position ofthe cursor 302, the position of the cursor 302 is not moved. As aresult, the operator 102 can operate without feeling strangeness, andreduction of operationality can be suppressed when the shape of the handof the operator is changed.

Second Exemplary Embodiment

According to the first exemplary embodiment, the cursor positioncorrecting unit 209 calculates cursor-position correction informationcpc (offset_x, offset_y), and the cursor control unit 211 corrects theposition of the cursor 302 based on the cursor-position correctioninformation cpc in a coordinate system of the display screen 301. Thesecond exemplary embodiment on the other hand is different from thefirst exemplary embodiment in that the cursor control unit 211 correctsthe cursor position by using a mapping expression mj for calculating acorrected position in a coordinate system of the operation range 103.Differences from the first exemplary embodiment will be described below.

FIG. 15 illustrates a gesture/offset correspondence table to be used bythe cursor position correcting unit 209 according to the secondexemplary embodiment. FIG. 16 illustrates a mapping expression table tobe used by the cursor position correcting unit 209 according to thesecond exemplary embodiment. The cursor position correcting unit 209calculates cursor-position correction information cpc describing acorrection amount for the position of the finger 106 in a coordinatesystem of the operation range 103 based on the positional information hpof the hand and positional information fp of the finger. The cursorposition correcting unit 209 then calculates the corrected position ofthe finger 106 in the coordinate system of the operation range 103 basedon a mapping expression mj corresponding to gesture information gesincluded in the mapping expression table and notifies the calculatedposition to the cursor control unit 211.

The cursor control unit 211 calculates cursor positional information cpbased on the corrected position of the finger 106 in the coordinatesystem of the operation range 103 notified from the cursor positioncorrecting unit 209 and a ratio of the number of pixels of the operationrange 103 to the number of pixels of the display screen 301. The cursorcontrol unit 211 then outputs the cursor positional information cp tothe drawing unit 212.

FIG. 17 is a flowchart illustrating an operating procedure correspondingto FIG. 11 according to the second exemplary embodiment. The processingin step S21 to S26 in FIG. 17 is identical to the processing in theflowchart illustrated in FIG. 11.

After the operation identifying unit 210 identifies the type ofoperation in step S26, the cursor position correcting unit 209 generatescursor-position correction information cpc (offset_x, offset_y) in thecoordinate system of the operation range 103 based on positionalinformation hp of the hand, positional information fp of the finger, andthe gesture/offset correspondence table by using the followingexpression (S41).

offset_x=xi−xp

offset_y=yi−yp

For example, when the positional information hp of the hand is (40, 15),and the tip positional information of the index finger is (60, 50),offset_x=20 and offset_y=35 are acquired.

Next, the cursor position correcting unit 209 calculates a correctedposition mpoint in the coordinate system of the operation range 103 byusing a mapping expression corresponding to the gesture information gesidentified by the gesture identifying unit 208 on the mapping expressiontable (S42).

mpoint_x=xi×(1−offset_x÷xi)

mpoint_y=yi×(1−offset_y÷yi)

In this case, when xi=60 and offset_x=20 are provided, mpoint_x=40 isacquired. When yi=50 and offset_y=35 are provided, mpoint_y=15 isacquired.

The cursor position correcting unit 209 notifies the calculated mpointvalue to the cursor control unit 211. The cursor control unit 211calculates the cursor positional information cp (xc, yc) based on thenotified mpoint value by using the following expressions to determinethe cursor position (S43).

xc=mpoint_x×the number of horizontal pixels of the display screen301÷the number of horizontal pixels of opr

yc=mpoint_y×the number of vertical pixels of the display screen 301÷thenumber of vertical pixels of opr

When mpoint_x=40 and mpoint_y=15 are provided, and the number ofhorizontal pixels of the display screen 301 is equal to 1920, the numberof vertical pixels of the display screen 301 is equal to 1080, thenumber of horizontal pixels of opr is 160, and the number of verticalpixels of opr is equal to 90, xc=480 and yc=180 are acquired. Thus, thecursor positional information cp is (480, 180). Therefore, also in thiscase, the position of the cursor 302 after the operator 102 stretchesthe finger 106 is not changed from the position of the cursor 302 beforethe operator 102 stretches the finger 106, like the first exemplaryembodiment.

The cursor control unit 211 may map the position in the operation range103 to the position in the display screen 301 by multiplying anothercoefficient, instead of the ratio of the number of pixels of the displayscreen 301 to the number of pixels of the operation region 103, forcalculating the cursor positional information cp (xc, yc). For example,the cursor control unit 211 may multiply mpoint by a coefficient havinga value decreasing as the distance from the corrected position mpoint inthe operation range 103 to the boundary decreases to calculate thecursor positional information cp (xc, yc). Thus, also when the hand ofthe operator 102 approaches to the boundary of the operation range 103,the cursor 302 can be moved within the display screen 301.

Effects of Second Exemplary Embodiment

As described above, also according to the second exemplary embodiment,when an operation based on a change in shape of a predetermined hand isperformed, the position of the cursor 302 does not move. Thus, theoperator 102 can operate display control apparatus 200 without feelingstrangeness and reduction of operationality can be suppressed when theshape of the hand of the operator is changed, like the first exemplaryembodiment. The corrected position mpoint in the coordinate system ofthe operation range 103 may be mapped to the position within the displayscreen 301 by using predetermined expressions as in the second exemplaryembodiment so that the display control apparatus 200 can flexiblycontrol the relationship between the position of the hand of theoperator 102 and the position of the cursor 302.

Having described the present invention with reference to embodiments,the technical scope of the present invention is not limited to the scopeof the embodiments, and various changes, modifications, and alterationsmay be made within the spirit and scope of the present invention.

For example, having described above that a CPU may execute the displaycontrol program to identify the shape and position of a hand based on acaptured image, embodiments of the present invention are not limitedthereto. A CPU may execute the display control program to acquireinformation describing the shape and position of a hand identified byexecuting another program and thus to identify the shape and position ofthe hand. The display control apparatus 200 may not have the capturedimage acquiring unit 204 and hand/finger identifying unit 206 but mayacquire information describing the shape and position of a hand from anexternal apparatus.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-160975, filed Aug. 18, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A display control apparatus comprising: anoperation identifying unit configured to identify a type of operationbased on a shape of a hand of an operator; and a display controllingunit configured to display an instruction image at an in-screen positioncorresponding to a first position indicated by the hand having a firstshape in a case where a first operation performed by the hand having thefirst shape is detected and display the instruction image at anin-screen position corresponding to a second position indicated by thehand having a second shape in a case where a second operation performedby the hand having the second shape is detected, wherein the displaycontrolling unit corrects the in-screen position of the instructionimage based on a relationship between the first position and the secondposition in a case where the second operation is detected after thefirst operation is detected and displays the instruction image at thein-screen position corresponding to the second position.
 2. The displaycontrol apparatus according to claim 1, wherein the display controllingunit displays the instruction image at the in-screen positioncorresponding to a third position in a case where the third operationperformed by a hand having a third shape indicating a third positiondifferent from the first position is detected after the first operationis detected without correcting the in-screen position of the instructionimage based on a relationship between the first position and the thirdposition.
 3. The display control apparatus according to claim 1, whereinthe first shape is a shape in which the operator does not stretch aspecific finger; and the second shape is a shape in which the operatorstretches the specific finger.
 4. The display control apparatusaccording to claim 1, wherein the display controlling unit corrects thein-screen position of the instruction image in a case where the secondoperation is detected after the first operation is detected so as toprevent the in-screen position of the instruction image from moving dueto a change from the first position to the second position.
 5. Thedisplay control apparatus according to claim 1, further comprising: anoperation-range identifying unit configured to acquire a captured imageincluding the hand of the operator and identify an operation range inwhich the operator moves the hand, wherein the display controlling unitcorrects the in-screen position corresponding to the second positionwith an in-screen difference corresponding to a difference between thefirst position and the second position to correct the in-screen positionof the instruction image, the difference being determined based on aratio of the number of pixels of the identified operation range to thenumber of pixels of the screen.
 6. The display control apparatusaccording to claim 5, wherein the display controlling unit displays asecond instruction image corresponding to a second hand at an in-screenposition corresponding to the position corrected from a position of thesecond hand based on the relationship in a case where the operationrange includes the second hand.
 7. The display control apparatusaccording to claim 1, wherein the display controlling unit continues tocorrection the in-screen position of the instruction image based on therelationship in a case where the hand having the second shape is movedafter the second operation is detected and moves the instruction imagein accordance with a movement of the hand having the second shape. 8.The display control apparatus according to claim 1, wherein the displaycontrolling unit displays the instruction image at an in-screen positioncorresponding to a third position different from the first position in acase where the first operation performed by the hand having the firstshape indicating the third position is detected after the instructionimage is displayed based on the second operation.
 9. The display controlapparatus according to claim 1, wherein the operation identifying unitidentifies the shape of the hand based on open/closed states of aplurality of fingers of the operator.
 10. The display control apparatusaccording to claim 1, wherein the display controlling unit displays onthe screen a display image corresponding to a position where theinstruction image is displayed and a type of the second operation in acase where the second operation is detected.
 11. A display controlmethod comprising: identifying a type of operation based on a shape of ahand of an operator; a first display step for displaying an instructionimage at an in-screen position corresponding to a first positionindicated by the hand having a first shape in a case where a firstoperation performed by the hand having the first shape is detected; andcorrecting the in-screen position of the instruction image based on arelationship between the first position and a second position differentfrom the first position indicated by the hand having a second shapeindicating the second position and a second display step for displayingthe instruction image at an in-screen position corresponding to thesecond position in a case where a second operation performed by the handhaving the second shape is detected after the first display step. 12.The display control method according to claim 11, the second displaystep includes displaying the instruction image at the in-screen positioncorresponding to a third position in a case where the third operationperformed by a hand having a third shape indicating a third positiondifferent from the first position is detected, without correcting thein-screen position of the instruction image based on the relationshipbetween the first position and the third position.
 13. The displaycontrol method according to claim 11, wherein the first shape is a shapein which the operator does not stretch a specific finger; and the secondshape is a shape in which the operator stretches the specific finger.14. The display control method according to claim 11, wherein the seconddisplay step includes correcting the in-screen position of theinstruction image in a case where the second operation is detected so asto prevent the in-screen position of the instruction image from movingdue to a change from the first position to the second position.
 15. Thedisplay control method according to claim 11, further comprisingacquiring a captured image including the hand of the operator andidentifying an operation range in which the operator moves the hand,wherein the second display step includes correcting the in-screenposition corresponding to the second position with an in-screendifference corresponding to a difference between the first position andthe second position to correct the in-screen position of the instructionimage, the difference being determined based on a ratio of the number ofpixels of the identified operation range to the number of pixels of thescreen.
 16. The display control method according to claim 15, furthercomprising displaying a second instruction image corresponding to asecond hand at an in-screen position corresponding to the positioncorrected from a position of the second hand based on the relationshipin a case where the operation range includes the second hand.
 17. Thedisplay control method according to claim 11, further comprisingcontinuing to correction the in-screen position of the instruction imagebased on the relationship in a case where the hand having the secondshape is moved after the second display step and moving the instructionimage in accordance with a movement of the hand having the second shape.18. The display control method according to claim 11, further comprisingdisplaying the instruction image at an in-screen position correspondingto a third position different from the first position in a case wherethe first operation performed by the hand having the first shapeindicating the third position is detected after the second display step.19. The display control method according to claim 11, further comprisingdisplaying on the screen a display image for executing a processcorresponding to a position where the instruction image is displayed anda type of the second operation after the second display step.
 20. Acomputer-readable recording medium storing a program for causing acomputer to execute: identifying a type of operation based on a shape ofa hand of an operator; a first display step for displaying aninstruction image at an in-screen position corresponding to a firstposition indicated by the hand having a first shape in a case where afirst operation performed by the hand having the first shape isdetected; and correcting the in-screen position of the instruction imagebased on a relationship between the first position and a second positiondifferent from the first position indicated by the hand having a secondshape indicating the second position and a second display step fordisplaying the instruction image at an in-screen position correspondingto the second position in a case where a second operation performed bythe hand having the second shape is detected after the first displaystep.